Non-transitory storage medium storing instructions executable by communication terminal, communication terminal, and processing apparatus

ABSTRACT

A non-transitory storage medium stores instructions executable by a communication terminal. The communication terminal includes: a display; an input device for detecting an instructed point where an instruction is provided with an input object; and a communication device for communicating with a processing apparatus. When executed by the computer, the instructions cause the communication terminal to: receive, from the processing apparatus, instruction image data for displaying an instruction image for instructing a processing to be executed by the processing apparatus; display the instruction image on the display based on the received instruction image data; and transmit, to the processing apparatus, an input event on the input device and coordinate information about the instructed point on the display which is detected by the input device in a state in which the instruction image is displayed.

CROSS REFERENCE TO RELATED APPLICATION

The present application claims priority from Japanese Patent ApplicationNos. 2013-203967 filed on Sep. 30, 2013, and 2013-203971 filed on Sep.30, 2013, the disclosures of which are herein incorporated by referencein its entirety.

BACKGROUND

Technical Field

The present invention relates to a non-transitory storage medium storinga plurality of instructions executable by a computer of a communicationterminal, the communication terminal, and a processing apparatus.

Description of the Related Art

A technique of remote control for controlling a multi-functionperipheral (MFP) with a mobile terminal including a touch panelsuperposed on an LCD has been proposed, for example. Specifically, theLCD of the mobile terminal displays an image acquired by taking apicture of an operation unit of the MFP, and the MFP executesprocessings in response to an operation on the touch panel of the mobileterminal. That is, when a user has touched an image corresponding to akey assigned with, e.g., a copy processing among images displayed on theLCD, the mobile terminal instructs the MFP to execute the copyprocessing.

SUMMARY

The above-described technique causes problems that a function forinstructing the MFP to execute a processing in response to an inputoperation on the touch panel of the mobile terminal needs to be createdfor each MFP and that functions need to be added or changed upon anupdate of the MFP.

This invention has been developed to provide (i) a non-transitorystorage medium storing a plurality of instructions executable by acomputer of a communication terminal including: a display device havinga display region; an input device configured to detect an instructedpoint which is a point on the display region and at which an instructionis provided with an input object; a communication device configured tocommunicate with a processing apparatus; and a controller, (ii) thecommunication terminal, and (iii) the processing apparatus, each of thenon-transitory storage medium, the communication terminal, and theprocessing apparatus being capable of providing versatility of afunction of controlling the processing apparatus based on an input onthe input device.

The present invention provides a non-transitory storage medium storing aplurality of instructions executable by a computer of a communicationterminal. The communication terminal includes: a display; an inputdevice configured to detect an instructed point which is a point on thedisplay and at which an instruction is provided with an input object;and a communication device configured to communicate with a processingapparatus. When executed by the computer, the plurality of instructionscause the communication terminal to perform: receiving instruction imagedata from the processing apparatus, the instruction image data being fordisplaying, on the display, an instruction image for instructing aprocessing to be executed by the processing apparatus; displaying theinstruction image on the display based on the received instruction imagedata; identifying an input event on the input device in response to aninstruction provided on the input device with the input object in astate in which the instruction image is displayed; and transmitting, tothe processing apparatus, the identified input event and coordinateinformation about the instructed point on the display which is detectedby the input device in a state in which the instruction image isdisplayed.

The present invention also provides a communication terminal, including:a display; an input device configured to detect an instructed pointwhich is a point on the display and at which an instruction is providedwith an input object; a communication device configured to communicatewith a processing apparatus; and a controller. The controller isconfigured to: receive instruction image data from the processingapparatus, the instruction image data being for displaying, on thedisplay, an instruction image for instructing a processing to beexecuted by the processing apparatus; display the instruction image onthe display based on the received instruction image data; identify aninput event on the input device in response to an instruction providedon the input device with the input object in a state in which theinstruction image is displayed; and transmit, to the processingapparatus, the identified input event and coordinate information aboutthe instructed point on the display which is detected by the inputdevice in a state in which the instruction image is displayed. Thecontroller is configured to execute a control using at least one of asoftware processing and a hardware processing.

The present invention also provides a processing apparatus controllableby a communication terminal. The communication terminal includes: adisplay; an input device configured to detect an instructed point whichis a point on the display and at which an instruction is provided withan input object; and a communication device configured to communicatewith the processing apparatus. The processing apparatus includes acontroller configured to: transmit instruction image data to thecommunication terminal, the instruction image data being for displaying,on the display, an instruction image for instructing a processing to beexecuted by the processing apparatus; receive, from the communicationterminal, an input event on the input device and coordinate informationabout the instructed point on the display which is detected by the inputdevice; and execute a processing based on the received input event andcoordinate information.

BRIEF DESCRIPTION OF THE DRAWINGS

The objects, features, advantages, and technical and industrialsignificance of the present invention will be better understood byreading the following detailed description of the embodiments of theinvention, when considered in connection with the accompanying drawings,in which:

FIG. 1 is a block diagram illustrating a system according to a firstembodiment;

FIGS. 2A-2C are views for explaining a processing for switching ascreen;

FIG. 3 is a table illustrating an image data map;

FIG. 4 is a flow chart illustrating a procedure of processings to beexecuted by the mobile terminal;

FIG. 5 is a view illustrating operations when an application isactivated in the mobile terminal;

FIG. 6 is a flow chart illustrating a procedure of an image sizingprocessing;

FIGS. 7A and 7B are views illustrating a processing for producingcoordinate information;

FIG. 8 is a flow chart illustrating a procedure of an event reactionprocessing;

FIG. 9 is a flow chart illustrating a procedure of processings to beexecuted by the MFP;

FIG. 10 is a flow chart illustrating a procedure of an event reactionprocessing in a second embodiment;

FIG. 11 is a flow chart illustrating a procedure of processings to beexecuted by the MFP in a third embodiment;

FIG. 12 is a flow chart illustrating a procedure of processings to beexecuted by the mobile terminal;

FIG. 13 is a flow chart illustrating a procedure of processings to beexecuted by the MFP in a fourth embodiment;

FIG. 14 is a flow chart illustrating a procedure of processings to beexecuted by the mobile terminal; and

FIGS. 15A and 15B are views each illustrating an image displayed on theMFP in a fifth embodiment.

DETAILED DESCRIPTION OF THE EMBODIMENTS

<First Embodiment>

Hereinafter, there will be described a first embodiment of the presentinvention by reference to the drawings.

FIG. 1 illustrates a system configuration in the present embodiment. Amulti-function peripheral (MFP) 10 has various functions such as aprinting function, a scanning function, and a facsimile function. TheMFP 10 includes a CPU 12, a flash memory 14, a RAM 16, a communicationdevice 18, a liquid crystal display (LCD) 20, a touch panel 22, ascanner 24, a printer 26, and a modem 28 which are connected to eachother via input/output ports 30. The CPU 12 controls the functions ofthe MFP 10 and the devices connected to the input/output ports 30according to fixed values and programs stored in the flash memory 14 anddata stored in the RAM 16.

The flash memory 14 is a rewritable non-transitory memory, and the RAM16 is a rewritable transitory memory. The LCD 20 is a liquid crystaldisplay for displaying various kinds of screens thereon. The touch panel22 is superposed on a display surface of the LCD 20. The touch panel 22is a user interface for outputting instructed point information whichindicates a location on the touch panel 22 at which an input object suchas a finger and a stylus has touched or approached the touch panel 22.The location or point at which the input object has touched orapproached the touch panel 22 may be hereinafter referred to as“instructed point”.

The touch or approach of the input object on the touch panel 22 may behereinafter referred to as “press of the input object on the touch panel22”, and an increase in distance between the input object and the touchpanel 22 after pressing may be hereinafter referred to as “leaving”.Also, an operation of moving or displacing the instructed point bymoving the input object without leaving it after pressing may behereinafter referred to as “drag operation”. In the present embodiment,the instructed point is a center of an area on the touch panel 22, whicharea is an area where the touch or approach of the input object has beendetected by the touch panel 22. However, the instructed point may beanother location. For example, the instructed point may be any point onthe area on the touch panel 22, or the area itself. These definitionsmay apply to a touch panel 46 of a mobile terminal 40.

The scanner 24 reads a document to convert a read image to image data.The printer 26 records an image on a recording sheet based on imagedata. The modem 28 upon facsimile transmission converts image data to betransmitted, into a signal transmittable to a telephone line network,not shown, and transmits the signal to the telephone line network. Also,the modem 28 receives a signal from the telephone line network to decodethe signal into image data. The communication device 18 is an interfacefor carrying out wireless communication. In the present embodiment, thewireless communication carried out by the communication device 18 isWi-Fi™ communication adhering to the IEEE802.11b/g standard.

The mobile terminal 40 is a device such as a smartphone and a mobilephone. The mobile terminal 40 includes a communication device 42, an LCD44, the touch panel 46, a voice input/output device 48, a telephonenetwork communication device 50, a CPU 52, a flash memory 54, and a RAM56 which are connected to each other via an input/output ports 58. TheCPU 52 controls the devices connected to the input/output ports 58according to fixed values and programs stored in the flash memory 54,for example.

The flash memory 54 is a rewritable non-transitory memory, and the RAM56 is a rewritable transitory memory. The LCD 44 is a liquid crystaldisplay for displaying various kinds of screens thereon. The touch panel46 superposed on a display surface of the LCD 44 is a user interface foroutputting instructed point information which indicates a location onthe touch panel 46 at which an input object such as a finger and astylus has touched or approached the touch panel 46.

The voice input/output device 48 is constituted by, e.g., a microphoneand a speaker. The telephone network communication device 50 is acircuit for conversation over a mobile phone network, not shown. Thecommunication device 42 is an interface for carrying out wirelesscommunication. In the present embodiment, the wireless communicationcarried out by the communication device 42 is Wi-Fi™ communicationadhering to the IEEE802.11b/g standard. The mobile terminal 40 isconnected to the MFP 10 including the communication device 18 throughthe Wi-Fi communication in an infrastructure mode in which connection isestablished via the communication device 42 and an access point, notshown, as a relay device.

The MFP 10 displays images for guiding input operations using the inputobject, on a display region 20 a of the LCD 20 and executes variouskinds of processings such as a facsimile transmission processing, aprint processing, and a scan processing in response to an inputoperation performed on the touch panel 22 such as pressing, dragging,and leaving. However, since the display region 20 a of the LCD 20 is notso large, all the images including objects (images) for guiding theinput operations for instructing execution of the processings cannot bedisplayed on the display region 20 a at a time. To address this problem,the MFP 10 in the present embodiment can switch an image displayed onthe display region 20 a of the LCD 20. FIG. 2A illustrates one exampleof an image displayed on the display region 20 a of the LCD 20, and FIG.2C illustrates another example of the image displayed on the displayregion 20 a of the LCD 20. In FIG. 2A, the images displayed include: anobject for designating the facsimile function (i.e., the Fax button); anobject for designating the copying function (i.e., the Copy button); andan object for designating the scanning function (i.e., the Scan button).In FIG. 2C, the images displayed include objects for instructing variouskinds of processings related to the facsimile function such as an objectfor instructing execution of the facsimile function (i.e., the Fax Startbutton).

In the present embodiment, when the image displayed on the displayregion 20 a of the LCD 20 is switched on the MFP 10, the MFP 10 changesthe image using an animation display in which the image is displaced atsuch a speed that a user can visually recognize at least positions ofthe objects on the image displayed on the display region 20 a. That is,in a case where the image displayed on the display region 20 a isswitched from the image illustrated in FIG. 2A to the image illustratedin FIG. 2C, the image (the entire screen) to be displayed after theswitch is gradually slid, during the switch of the image, from the rightside of the image displayed before the switch (i.e., the right side ofthe entire screen). FIG. 2B illustrates a screen on which the image isbeing slid. In this period, the touch panel 22 of the MFP 10 does notaccept or receive the input operation.

The image displayed on the display region 20 a of the LCD 20 is switchedfundamentally in response to the input operation performed by the user.Specifically, the image is switched when the input object has left thetouch panel 22 after the user had pressed the object for triggeringscreen transition on the touch panel 22. That is, in the presentembodiment, pressing on the touch panel 22 means an operation ofchoosing the location of the instructed point, and the leaving operationmeans an operation of confirming the location of the instructed point.It is noted that the location of the pressed instructed point is storedinto the RAM 16.

In the present embodiment, when information which should be notified tothe user is generated in the MFP 10, the screen on the display region 20a of the LCD 20 may be switched without the leaving operation on thetouch panel 22, i.e., without pressing an object for triggering screentransition and leaving the touch panel 22. Here, the information whichshould be notified to the user is generated when a remaining amount ofink in the printer 26 becomes low, for example. This screen transitionis fundamentally provided as a pop-up display. It is noted that thepop-up display is one kind of display in which an image appears on thedisplay region 20 a for a predetermined length of time and thendisappears.

In the present embodiment, each screen to be displayed on the displayregion 20 a of the LCD 20 is assigned with information or a screen IDfor identifying the screen. FIG. 3 illustrates an image data map storedin the flash memory 14 of the MFP 10, and this image data map representsa relationship between each screen ID and a corresponding image.Specifically, for each image data, the image data map represents arelationship among the image data, the screen ID, and information ofwhether storing is required or not. The information of whether storingis required or not will be explained later in detail in explanation of aprocessing at S120 in FIG. 9.

The MFP 10 can execute the functions not only in response to the inputoperations on the touch panel 22 but also in response to instructionstransmitted from the mobile terminal 40. To enable this function, theLCD 44 of the mobile terminal 40 can display the image displayed on thedisplay region 20 a for guiding the input operation using the inputobject in the present embodiment. That is, the MFP 10 sends the mobileterminal 40 image data representative of this image, and the mobileterminal 40 displays an image based on the received image data on adisplay region 44 a of the LCD 44. This configuration allows the user touse the input object to perform the input operation on the touch panel46 according to the image displayed on the display region 44 a of theLCD 44. When the input operation is performed by the user, the mobileterminal 40 sends the MFP 10 coordinate information about the instructedpoint at which the input operation is performed on the display region 44a of the touch panel 46 of the LCD 44. As explained later in detail withreference to FIG. 7, the coordinate information allows the MFP 10 torecognize which position is instructed on the image displayed on thedisplay region 20 a of the LCD 20. For example, in a case where theinstructed point is located on the object displayed on the displayregion 44 a for instructing execution of the print processing, the MFP10 based on the coordinate information recognizes the instruction ofexecution of the print processing and executes the print processing.Thus, the user can operate the MFP 10 by performing the input operationon the touch panel 46 of the mobile terminal 40. In this operation, themobile terminal 40 only notifies the MFP 10 of the coordinateinformation about the instructed point and a type of an input event onthe touch panel 46 (e.g., pressing, dragging, or leaving), eliminating aneed for the mobile terminal 40 to recognize which kind of processing ofthe MFP 10 is instructed by the input operation on the touch panel 46.

There will be next explained, with reference to drawings, processingsexecuted when operations of the MFP 10 are instructed from the mobileterminal 40.

FIG. 4 illustrates a procedure of processings to be executed by themobile terminal 40. The CPU 52 executes these processings by executingan operation support program stored in the flash memory 54. It is notedthat processings at S14-S54 are executed each time when the CPU 52 atS14 accepts the input operation on the touch panel 46. That is, theprocessings at S14-S54 can be executed in parallel.

When the operation support program or application is activated, thisflow begins with S10 at which the CPU 52 executes an LCD data acquiringprocessing. That is, the CPU 52 as illustrated in FIG. 5 sends the MFP10 a signal for requesting the image data representative of the imagedisplayed on the display region 20 a of the LCD 20 of the MFP 10 (notedthat this data may be hereinafter referred to as “LCD data”). When theMFP 10 transmits the LCD data to the mobile terminal 40, the CPU 52receives the LCD data and displays an image on the display region 44 aof the LCD 44 based on the LCD data.

The CPU 52 at S12 receives supplemental information from the MFP 10.Here, the supplemental information contains a screen ID for identifyingthe image requested at S10. The MFP 10 uses the image data mapillustrated in FIG. 3 to incorporate a screen ID corresponding to theimage data transmitted to the mobile terminal 40, into the supplementalinformation.

The CPU 52 at S14 waits for the input operation on the touch panel 46.When the CPU 52 accepts the input operation, the CPU 52 at S16determines whether the input event relating to the input operation is apress event or not. When the input event is the press event (S16: Yes),the CPU 52 stores the location of the pressed instructed point into theRAM 56 and at S18 determines whether the input event on the touch panel46 is acceptable or not. This processing is for determining whether aprocessing at S46 which will be described below is being executed ornot. When the input event on the touch panel 46 is not acceptable (S18:No), this flow goes to S14 to accept the input event when the inputoperation on the touch panel 46 becomes acceptable. When the input eventon the touch panel 46 is acceptable (S18: Yes), the CPU 52 executes animage sizing processing at S20.

FIG. 6 illustrates a procedure of the image sizing processing. Thisprocessing is for distinguishing whether the press event accepted at S14is the press event for instructing the MFP 10 to execute the processingor one of successive press events accepted when a pinch-in operation ora pinch-out operation is performed using the input objects. Here, thepinch-in operation is an operation in which a thumb and an index finger,for example, touching the touch panel 46 are moved to each other, andthis pinch-in operation is performed to reduce the size of the image.The pinch-out operation is an operation in which the thumb and the indexfinger, for example, touching the touch panel 46 are moved away fromeach other, and this pinch-out operation is performed to increase thesize of the image. Since the pinch-in operation and the pinch-outoperation are respectively for instructions of reducing and increasingthe size of the image displayed on the image displayed on the displayregion 44 a of the LCD 44 of the mobile terminal 40, the pinch-inoperation and the pinch-out operation are not related to the MFP 10.

The CPU 52 at S60 determines whether a wait processing explained at S61has been executed or not. When the wait processing has not been executed(S60: No), the CPU 52 at S61 starts the wait processing in which the CPU52 waits for a predetermined length of time. Here, in a case where theuser presses the touch panel 46 with two fingers as the input objects toperform the pinch-in operation or the pinch-out operation, thepredetermined length of time is set according to a difference in timebetween a pair of pressing timings. Specifically, the predeterminedlength of time is set at a value normally taken as the above-describeddifference in time. For example, the predetermined length of time is setat 0.5 seconds in the present embodiment. The CPU 52 at S62 determineswhether another press event has occurred during the wait processing ornot. This processing is for determining whether or not the pinch-inoperation or the pinch-out operation is to be performed. When anotherpress event has occurred during the wait processing (S62: Yes), the CPU52 at S64 executes a processing for increasing or reducing the size ofthe image displayed on the display region 44 a of the LCD 44, accordingto movement of a pair of instructed points corresponding to the pair ofpress events. In reality, the processing at S62 is not preferablyexecuted after waiting for the predetermined length of time, and theprocessing at S62 is preferably repeated until a positive decision ismade before the predetermined length of time has passed. With thisconfiguration, when a positive decision is made at S62, this flow canspeedily go to S64.

When another press event has not occurred during the wait processing(S62: No), this flow goes to S22 in FIG. 4.

When a positive decision is made again at S16 in FIG. 4 by a touch orapproach of another input object on or toward the touch panel 22, andthe flow goes to the image sizing processing before a leaving event hasoccurred, the CPU 52 determines that the wait processing at S61 is beingexecuted (S60: Yes), and the flow goes to S72. There are two instructedpoints at this time, and accordingly the user may intend to perform thepinch-in operation or the pinch-out operation. However, since twoinstructed points are established during the wait processing, the CPU 52at S72 distinguishes whether the input operation is the pinch-inoperation or the pinch-out operation, or an accidental press of thetouch panel 46 with another finger during a drag.

Specifically, the CPU 52 at S72 determines whether or not the number ofstorages of drags which is incremented at S36 which will be describedbelow is smaller than or equal to a predetermined number. The number ofstorages of drags indicates the number of samplings in a case where thedrag operation is being performed and indicates that a length of time ofthe drag is longer with increase in the number of storages of drags.Here, the predetermined number is set at a value that is difficult toassume in the case of the pinch-in operation or the pinch-out operation.When the number of storages of drags is larger than the predeterminednumber (S72: No), the CPU 52 determines that the length of time of thedrag operation is too long for the pinch-in operation or the pinch-outoperation, and this flow goes to S22 in FIG. 4. When the number ofstorages of drags is smaller than or equal to the predetermined number(S72: Yes), the CPU 52 at S74 notifies the MFP 10 of a cancel oftransmission of the press event since it is considered that the pinch-inoperation or the pinch-out operation is to be performed. This cancel isperformed because when the positive decision is made at S72, the userhas intended to perform the pinch-in operation or the pinch-outoperation, but it has taken a relatively long time from detection of oneinstructed point on the touch panel 46 to detection of another oneinstructed point on the touch panel 46, and accordingly the negativedecision is made at S62, and after the processing at S22 which will bedescribed below, the press event has been transmitted to the MFP 10 atS24 which will be described below. This canceling processing is aprecondition to the processing at S160 in FIG. 9 which will be describedbelow. Upon completion of the processing at S74, the CPU 52 at S64executes the sizing processing according to movement of the pair ofinstructed points.

After the execution of the processing at S64, the CPU 52 at S66determines whether the touch panel 46 has ceased detecting all theinstructed points or not. In other words, the CPU 52 determines whetherall the input objects have left the touch panel 22 or not. When thetouch panel 46 has ceased detecting all the instructed points (S66: No),the CPU 52 at S68 initialize the number of storages of drags, and thisflow returns to S14 in FIG. 4 to wait for a new input on the touch panel46. In reality, the number of storages of drags at S68 changes to theinitialized value in a case where the CPU 52 executes the processing atS74, and when the positive decision is made at S62, the CPU 52 does notneed to execute the processing at S68.

When the processing at S20 in FIG. 4 is completed, that is, when anegative decision is made at S62 or S72, the CPU 52 at S22 converts apressed location. For example, as illustrated in FIG. 7A, the CPU 52obtains a value {(A−X)/k, (B−Y)/k} by dividing a difference (A−X, B−Y)between coordinates (drawing start coordinates: X, Y) on the displayregion 44 a of the LCD 44 which are used as a reference for display ofan image and coordinates (A, B) of the pressed point, by an enlargementratio k of the image being displayed. The difference (A−X, B−Y) iscoordinates of the pressed point provided in a case where the drawingstart coordinates (X, Y) are designated as an origin point. Here, theenlargement ratio k is a ratio of the size of the image displayed on thedisplay region 44 a of the LCD 44 of the mobile terminal 40 to the sizeof the image displayed on the display region 20 a of the LCD 20 of theMFP 10. It is noted that the size is not limited to the actual lengthand may be the number of pixels or screen resolutions, for example. Inthis case, in a case where the area of each pixel is smaller on the LCD44 of the mobile terminal 40 than on the LCD 20 of the MFP 10, forexample, even where the enlargement ratio is one, the actual size of theimage displayed is smaller on the LCD 44 than on the LCD 20.

The converting processing is for allowing the MFP 10 to recognize whichpoint on the image displayed on the display region 20 a of the LCD 20 isthe pressed point, even in the case where the enlarged or reduced-sizeimage is being displayed on the display region 44 a of the LCD 44 of themobile terminal 40. This function of the converting processing isachieved also based on, as illustrated in FIG. 7B, a ratio of the lengthfrom the reference point to the pressed point to the length of the imageon each component of the two-dimensional coordinates. That is, a pair ofend portions of a diagonal line on the image may be respectively definedas the drawing start coordinates (X, Y) and drawing end coordinates (Xe,Ye), and the function of the converting processing may be achieved basedon a ratio {(A−X)/(Xe−X), (B−Y)/(Ye−Y)} between the length of thedisplayed image and the length from the drawing start coordinates to thepressed point on each component of the two-dimensional coordinates.

At S24 in FIG. 4, the CPU 52 sends the MFP 10 pressed locationinformation obtained by the above-described processing (i.e., coordinateinformation about the point instructed by a press), informationindicating that the type of the input event is the press event, and thescreen ID of the image being displayed on the display region 44 a of theLCD 44. The CPU 52 at S26 executes an event reaction processing.

FIG. 8 illustrates this event reaction processing. In this eventreaction processing, the CPU 52 at S80 determines whether the CPU 52 hasreceived, from the MFP 10, a notification indicating that the screen IDof the image displayed on the display region 44 a of the LCD 44 of themobile terminal 40 differs from the screen ID of the image displayed onthe display region 20 a of the LCD 20 of the MFP 10. This notificationis transmitted in a processing at S140 in FIG. 9 (which will bedescribed below) executed by the MFP 10 according the processing at S24in FIG. 4. Here, the image displayed on the display region 20 a of theLCD 20 of the MFP 10 is not changed by transmission of the press eventto the MFP 10 in response to a press operation on the touch panel 46 ofthe mobile terminal 40. This is because, as described above, the screenis changed by the operation for confirming the instructed point (i.e.,the leaving operation) in the present embodiment. The screen IDs maydiffer from each other nonetheless because in the case where theinformation which should be notified to the user is generated in the MFP10, the image displayed on the display region 20 a of the LCD 20 isswitched independently of an input operation performed by the user asdescribed above.

When the notification is not received for a predetermined length of time(S80: No), the event reaction processing ends. When the notification isnot received (S80: Yes), the CPU 52 at S82 determines that there is LCDdata to be acquired from the MFP 10 and executes the LCD data acquiringprocessing. This processing is similar to the processing at S10 in FIG.4. The CPU 52 at S84 executes a supplemental information acquiringprocessing. This processing is generally similar to the processing atS12 in FIG. 4, but the CPU 52 at S84 acquires not only the screen ID butalso next screen presence/absence information. The CPU 52 at S86displays, on the display region 44 a of the LCD 44, an image indicatingthat a screen is to be updated because the screen IDs are different. Oneexample of this notification is a pop-up display. This processing isexecuted to notify the user of a switch of the screen which is causedbecause the image displayed on the display region 20 a of the LCD 20 ofthe MFP 10 differs from the image displayed on the display region 44 aof the LCD 44 of the mobile terminal 40.

The CPU 52 at S88 initializes the number of storages of drags which willbe described below. At S90, the CPU 52 determines, based on thesupplemental information received at S84, whether there is a next imageto be acquired from the MFP 10 (hereinafter may be simply referred to as“next image”) or not. This processing is on the precondition of asetting in which when an image displayed on the display region 20 a ofthe LCD 20 has been changed in the MFP 10 independently of the mobileterminal 40, the mobile terminal 40 successively receives image datarespectively representative of a plurality of images, and when theseimages should be displayed on the display region 44 a of the LCD 44,information indicating the presence of the next image is incorporatedinto the supplemental information. It is noted that this setting will beexplained later in detail at processings at S114-S122 in FIG. 9 whichwill be described below. When there is no next image (S90: No), theevent reaction processing ends. When there is a next image (S90: Yes),the CPU 52 at S92 executes the wait processing, and this flow returns toS82. Here, a waiting time in the wait processing is set at a time thatis long enough for the user to recognize the image being displayed onthe display region 44 a of the LCD 44.

It is noted that when the event reaction processing ends, this flowreturns to S14 in FIG. 4 to wait for a new input operation on the touchpanel 46.

When the newly occurred event is not the press event (S16: No), the CPU52 at S28 determines whether the newly occurred event is the drag eventor not. When the CPU 52 determines that the newly occurred event is thedrag event, based on detection of a displacement of the instructed pointon the touch panel 46 (S28: Yes), the CPU 52 at S30 determines whetherthe input event on the touch panel 46 is acceptable or not as in theprocessing at S18. When the input event on the touch panel 46 is notacceptable (S30: No), this flow returns to S14 to accept the input eventon the touch panel 46 in a case where the input event becomesacceptable. When the input event on the touch panel 46 is acceptable(S30: Yes), the CPU 52 at S32 executes a dragged location convertingprocessing. This processing is for converting the current location ofthe instructed point, to a value or coordinate information that can beidentified as a location on the image displayed on the display region 20a of the LCD 20 of the MFP 10, in the manner similar to that illustratedin FIG. 7A or 7B, regardless of the size of the image displayed on thedisplay region 44 a of the LCD 44.

The CPU 52 at S34 sends the MFP 10 (i) information indicating that thetype of the newly occurred event is a drag (i.e., the drag event) and(ii) coordinate information about the instructed point which is obtainedat S32. The CPU 52 at S36 executes a processing for storing, into theRAM 56, the number of drag events which indicates the number ofsamplings of the dragged locations at S32. This processing is aprecondition to the processing at S72 in FIG. 6. When the processing atS36 ends, this flow returns to S14 to wait for a new input operation onthe touch panel 46.

When the newly occurred event is not the drag event (S28: No), the CPU52 at S38 determines whether the newly occurred event is the leavingevent or not. When the newly occurred event is not the leaving event(S38: No), this flow returns to S14 to wait for a new input operation onthe touch panel 46. When the newly occurred event is the leaving event(S38: Yes), the CPU 52 at S40 determines whether the input event on thetouch panel 46 is acceptable or not as in the processing at S18. Whenthe input event on the touch panel 46 is not acceptable (S40: No), thisflow returns to S14 to accept the input event on the touch panel 46 in acase where the input event becomes acceptable. When the input event onthe touch panel 46 is acceptable (S40: Yes), the CPU 52 at S42 executesa left location converting processing. This processing is for convertinga location at which the instructed point has disappeared, in otherwords, the location of the instructed point detected most recently (thepressed location or the dragged location), to a value that can beidentified as a location on the image displayed on the display region 20a of the LCD 20 of the MFP 10, in the manner similar to that illustratedin FIG. 7A or 7B, regardless of the size of the image displayed on thedisplay region 44 a of the LCD 44.

The CPU 52 at S44 sends the MFP 10 (i) information indicating that thetype of the newly occurred event is leaving, (ii) the screen ID of theimage displayed on the LCD 44, and (iii) left location informationobtained at S42 (i.e., coordinate information about the instructed pointhaving disappeared upon leaving). The CPU 52 at S46 starts prohibitingan acceptance of the event and at S48 executes the wait processing. Uponcompletion of the wait processing, the CPU 52 at S50 executes an LCDdata acquiring processing similar to the processing at S10 and at S52executes a supplemental information acquiring processing similar to theprocessing at S12. It is noted that the CPU 52 at S52 acquires not onlythe screen ID but also the next screen presence/absence information. Thewait processing is a processing in which the CPU 52 waits fortransmission of a LCD-data requesting signal for a length of timerequired for the image displayed on the display region 20 a of the LCD20 of the MFP 10 to be switched in response to the leaving event. Thiswait processing is on the precondition that the animation display isperformed in the case where the image to be displayed on the displayregion 20 a of the LCD 20 is switched in the present embodiment asexplained above with reference to FIGS. 2A-2C.

If the mobile terminal 40 requests the MFP 10 to transmit the LCD datawithin a period of the animation display, the image displayed on the MFP10 and the image displayed on the mobile terminal 40 may unfortunatelydiffer from each other, and if the input operation is received in thiscase, a malfunction may occur in the MFP 10. That is, the LCD dataduring the animation is transmitted to the mobile terminal 40, and whena switch of the image displayed on the display region 20 a of the LCD 20of the MFP 10 is finished within a period in which an image based on thedata is displayed on the display region 44 a of the LCD 44, the inputoperation on the touch panel 22 is allowed in the MFP 10 at this time.If the input operation on the touch panel 46 is performed on the mobileterminal 40 at this time based on the image illustrated in FIG. 2B anddisplayed on the display region 44 a of the LCD 44, and coordinateinformation about the instructed point is transmitted to the MFP 10, theMFP 10 may erroneously recognize that the input operation is performedon the image displayed on the display region 20 a of the LCD 20 at alocation that differs from a location intended by the user. If theinstructed point erroneously recognized is located on the display region20 a of the LCD 20 at an object indicating a location at which aninstructed point should be positioned for instructing the processing tobe executed by the MFP 10, a malfunction may occur in the MFP 10. Toprevent this problem, the touch panel 46 of the mobile terminal 40 doesnot accept the input operation thereon within the period of theanimation display on the MFP 10 in the processings at S46 and S54 inFIG. 4.

Upon completion of the processing at S52, the CPU 52 at S54 cancels orfinishes the processing for prohibiting the acceptance of the event, andthis flow returns to S14 to wait for a new input operation on the touchpanel 46. It is noted that the reason why this processing is notexecuted at the completion of the processing at S48 is for canceling theacceptance prohibiting processing on condition that the image displayedon the display region 44 a of the LCD 44 is switched to the imagedisplayed on the display region 20 a of the LCD 20.

FIG. 9 illustrates a procedure of processings to be executed by the MFP10. These processings are executed by the CPU 12 of the MFP 10.

This flow begins with S100 at which the CPU 12 executes an event standbyprocessing for waiting for transmission of information such as the typeof event and the coordinate information from the mobile terminal 40.When an event occurs, the CPU 12 at S102 determines whether the event isa request for the LCD data or not. When the event is the request for theLCD data (S102: Yes), the CPU 12 at S104 determines whether there is LCDdata stored in the RAM 16 or not. This processing is provided in view ofthe above-described situation in which when the information which shouldbe notified to the user is generated in the MFP 10, the image displayedon the display region 20 a of the LCD 20 of the MFP 10 may be switchedregardless of the input operation performed on the touch panel 46 of themobile terminal 40. That is, since image data not having beentransmitted to the mobile terminal 40 yet is stored into the RAM 16 inthis case at a processing at S122 which will be described below, the CPU12 at S104 determines whether or not there is image data stored in theRAM 16 independently of the image data representative of the image beingdisplayed on the display region 20 a of the LCD 20. When there is no LCDdata stored in the RAM 16 (S104: No), the CPU 12 at S106 acquires ascreen ID that is an ID assigned to the image data displayed on thedisplay region 20 a of the LCD 20. This processing can be executed basedon the image data map illustrated in FIG. 3. The CPU 12 at S108 storesthe acquired screen ID into the RAM 16 as a transfer screen ID to betransmitted to the mobile terminal 40. The CPU 12 at S110 transfers theLCD data and the transfer screen ID as the supplemental information tothe mobile terminal 40. In reality, the processing for transferring thesupplemental information is executed when the supplemental informationis requested by the mobile terminal 40 at S12, S52, or S84 in FIGS. 4and 8 independently of the request for the LCD data.

When the event is not the LCD-data-acquisition instructing event (S102:No), the CPU 12 at S116 determines whether an event of a transition ofthe screen displayed on the LCD 20 has occurred or not. This processingis for determining whether or not the event of the transition of thescreen displayed on the LCD 20 has occurred despite the absence of theinstruction from the mobile terminal 40. One example of the screentransition in this case is a pop-up display which is caused when theinformation which should be notified to the user is generated in the MFP10 as described above. Here, the screen ID assigned to an imagecorresponding to the pop-up display differs from the screen ID assignedto an image not corresponding to the pop-up display in the image datamap illustrated in FIG. 3.

When the screen transition event has occurred (S116: Yes), the CPU 12 atS118 determines whether or not the screen ID of the image displayedbefore the switch coincides with or matches the transfer screen IDstored at S108. This processing is for determining whether or not theimage displayed before the switch has not been transferred to the mobileterminal 40 yet. When the screen ID of the image displayed before theswitch does not coincide with the transfer screen ID (S118: No), the CPU12 at S120 determines whether or not the image displayed before theswitch is a screen that needs to be stored. This processing is executedwith reference to the image data map illustrated in FIG. 3. That is, theneed for the storage is defined in advance in the image data map, andaccordingly the CPU 12 determines whether the CPU 12 needs to storeimage data representative of the image or not based on the image datamap. When the CPU 12 needs to store the image data representative of theimage (S120: Yes), the CPU 12 at S122 stores the LCD data into the RAM16. When the screen ID of the image displayed before the switchcoincides with the transfer screen ID after the storage of the LCD dataor at S118 (S118: Yes) or when the image displayed before the switch isnot a screen that needs to be stored (S120: No), the CPU 12 at S124executes a processing for screen transition, and this flow goes to S100to wait for a new event.

When the CPU 12 at S104 determines that there is any LCD data stored inthe RAM 16 (S104: Yes), the CPU 12 at S112 acquires the LCD data storedin the RAM 16. The CPU 12 at S114 transfers the LCD data and thesupplemental information, and this flow returns to S100 to wait for anew event. Here, the supplemental information contains informationindicating that there is a next image. This is a setting for causing themobile terminal 40 to transmit a signal requesting for the LCD dataagain. That is, in the present embodiment, the MFP 10 transfers the LCDdata to the mobile terminal 40 in response to the request for the LCDdata from the mobile terminal 40, and the MFP 10 never transfers the LCDdata to the mobile terminal 40 without receiving the request. Here, thesituation in which there is LCD data stored in the RAM 16 is a situationin which the negative decision is made at S118, and the positivedecision is made at S120, and this situation cannot be recognized by themobile terminal 40. Accordingly, the supplemental information containsthe information indicating that there is the next image. Consequently,when the positive decision is made at S90 in FIG. 8, the CPU 52 at S82requests the LCD data again.

When the newly occurred event is not the screen transition (S116: No),the CPU 12 at S126 determines whether the newly occurred event is thepress event or not. When the newly occurred event is the press event(S126: Yes), the CPU 12 at S128 acquires or receives a screen IDattached to the event. This processing is on the precondition of theprocessing at S24 in FIG. 4. The CPU 12 at S130 acquires the screen IDof the image displayed on the display region 20 a of the LCD 20. The CPU12 at S132 determines whether the acquired screen ID attached to theevent coincides with the screen ID of the image displayed on the displayregion 20 a of the LCD 20 or not. This processing is for determiningwhether the image displayed on the display region 20 a of the LCD 20 ofthe MFP 10 coincides with the image displayed on the display region 44 aof the LCD 44 of the mobile terminal 40 or not. When the acquired screenID attached to the event coincides with the screen ID of the imagedisplayed on the display region 20 a of the LCD 20 (S132: Yes), the CPU12 determines that the press event transmitted from the mobile terminal40 is effective as the instruction for causing the MFP 10 to execute aprocessing corresponding to the press event, and the CPU 12 executesprocessings at S134-S138. That is, the CPU 12 at S134 converts pressedlocation data if necessary.

The CPU 12 at S136 stores the pressed location into the RAM 16. Thisprocessing is a precondition to a processing at S164 which will bedescribed below. The CPU 12 at S138 executes a press processing which isa processing to be executed in response to the pressing, and this flowreturns to S100 to wait for a new event. In a case where there is anobject at a pressed location, for example, this processing creates adisplay for visually indicating that the object has been chosen. It isnoted that this processing may cause a processing only inside the MFP 10without causing any changes to the display region 20 a of the LCD 20.

When the CPU 12 at S132 determines that the acquired screen ID attachedto the event does not coincide with the screen ID of the image displayedon the display region 20 a of the LCD 20 (S132: No), the CPU 12 at S140notifies of the mobile terminal 40 about information indicating that thescreen IDs do not coincide with each other, and this flow returns toS100 to wait for a new event. The processing at S140 is a preconditionto the processing at S80 in FIG. 8. When the negative decision is madeat S132, the CPU 12 does not execute the press processing at S138. Thisis because, when the image displayed on the display region 20 a of theLCD 20 of the MFP 10 is not the same as the image displayed on thedisplay region 44 a of the LCD 44 of the mobile terminal 40, it isconsidered that the press event transmitted from the mobile terminal 40is not effective as the instruction for causing the MFP 10 to execute aprocessing corresponding to the press event.

When the newly occurred event is not the press event (S126: No), the CPU12 at S142 determines whether the newly occurred event is a draginstructing event or not. When the newly occurred event is the draginstructing event (S142: Yes), the CPU 12 at S144 converts draggedlocation data, if necessary, in the manner similar to the processing atS134. The CPU 12 at S146 determines whether or not the CPU 12 needs tocontrol the LCD 20 of the MFP 10 to cause a display based on the drag,e.g., a display for moving the dragged image, in response to the dragoperation on the touch panel 46 of the mobile terminal 40. The CPU 12can execute this processing by storing values into, e.g., the flashmemory 54 of the MFP 10 in advance, and each of these values determineswhether the CPU 12 needs to control the LCD 20 to cause a display basedon the drag or not. It is noted that the requirement of the displaybased on the drag may be determined at shipment of the MFP 10 and may beselectable by the user. When the image during the drag is required(S146: Yes), the CPU 12 at S148 controls the LCD 20 of the MFP 10 tocause a display based on the drag in response to the drag operation onthe touch panel 46 of the mobile terminal 40. When the image during thedrag is not required (S146: No), the CPU 12 at S150 stores, into the RAM16, the current instructed point moved after the drag, this flow returnsto S100 to wait for a new event.

When the newly occurred event is not the drag instructing event (S142:No), the CPU 12 at S152 determines whether the drag instructing event isa leaving instructing event or not. When the drag instructing event isthe leaving instructing event (S152: Yes), the CPU 12 at S154 determineswhether the location of the instructed point changed by the drag fromthe instructed point at the pressing is stored at S150 or not. When thechanged location of the instructed point is stored (S154: Yes), the CPU12 at S156 displaces the instructed point to the stored location on thedisplay region 20 a of the LCD 20. When this processing is completed orwhen the changed location of the instructed point is not stored (S154:No), the CPU 12 executes a leaving processing at S158, and this flowreturns to S100 to wait for a new event. It is noted that the leavingprocessing is executed on the MFP 10 in response to the leavingoperation performed on the touch panel 46 of the mobile terminal 40. Forexample, in a case where the leaving operation is performed on an objectdisplayed on the display region 44 a of the LCD 44 of the mobileterminal 40, and thereby the choice of the object is confirmed, adisplay based on this operation is caused on the display region 20 a ofthe LCD 20 of the MFP 10.

When the newly occurred event is not the leaving instructing event(S152: No), the CPU 12 at S160 determines whether or not the newlyoccurred event is a press cancel event transmitted at S74 in FIG. 6.When the newly occurred event is not the press cancel event (S160: No),this flow returns to S100 to wait for a new event. When the newlyoccurred event is the press cancel event (S160: Yes), the CPU 12 at S162determines whether or not a drag event has occurred just after thepressing operation. When the drag event has occurred (S162: Yes), theCPU 12 at S164 returns, to the pressed location, the location of theinstructed point on the LCD 20 and the location of the object displacedin response to the drag event. This processing is executed because thelocation of the displayed object is moved back to the pressed locationin the case where the location of the displayed object chosen by thepress is changed by the drag operation continued after the press, andthereafter the press event is canceled. When this processing iscompleted or when the drag event has not occurred (S162: No), the CPU 12executes a press canceling processing at S166, and this flow returns toS100 to wait for a new event. Here, in a case where the press processingis a display of information indicating that the pressed object ischosen, the display of the information is canceled in the presscanceling processing, for example.

The present embodiment described above provides the following effects.

The image displayed on the display region 20 a of the LCD 20 of the MFP10 is displayed on the display region 44 a of the LCD 44 of the mobileterminal 40. This configuration allows the user to perform the inputoperation on the touch panel 46 of the mobile terminal 40 in the sameway as the user performs the input operation on the touch panel 22 ofthe MFP 10.

The coordinate information about the instructed point (i.e., (x, y) inFIG. 7A or (m, n) in FIG. 7B) determined based on the input operationperformed on the touch panel 46 of the mobile terminal 40 is transmittedto the MFP 10 at S24, S34, or S44 in FIG. 4. This configurationeliminates a need for the mobile terminal 40 to recognize whichprocessing of the MFP 10 is instructed by the input operation on thetouch panel 46, thereby providing general versatility in the functionfor controlling the MFP 10 based on the input to the touch panel 46.

The screen ID is assigned to the image data (FIG. 3), and the mobileterminal 40 receives the screen ID upon receiving the LCD data andtransmits the screen ID upon transmitting the coordinate information andso on to the MFP 10. This configuration allows the MFP 10 to recognizewhether the image displayed on the display region 44 a of the LCD 44 ofthe mobile terminal 40 matches the image displayed on the display region20 a of the LCD 20 of the MFP 10. Accordingly, the MFP 10 can deal witha mismatch.

When the image displayed on the display region 44 a of the LCD 44 of themobile terminal 40 does not match the image displayed on the displayregion 20 a of the LCD 20 of the MFP 10 (S132: No), the mobile terminal40 requests the LCD data at S82. This configuration can eliminate themismatch.

When the image displayed on the display region 44 a of the LCD 44 of themobile terminal 40 does not match the image displayed on the displayregion 20 a of the LCD 20 of the MFP 10 (S132: No), the mobile terminal40 receives the LCD data from the MFP 10 to update the display and atS86 displays the information indicating that the image is updated toeliminate the mismatch. This configuration can notice the user of thesituation.

The mobile terminal 40 requests and receives the supplementalinformation containing the screen ID, and the LCD data separately. Withthis configuration, in a state in which the mobile terminal 40 canconvert the LCD data (e.g., PDF data) transmitted from the MFP 10, into,e.g., JPEG data in a state in which no data is attached to the LCD data.Accordingly, the mobile terminal 40 can convert the LCD data whencompared with a case where the supplemental information and the LCD dataare combined.

The mobile terminal 40 does not accept the input operation on the touchpanel 46 until the mobile terminal 40 receives the requested LCD dataand displays the image on the display region 44 a of the LCD 44 based onthe LCD data (specifically the processing at S14 is executed after theprocessing at S10 in FIG. 4, and the flow goes to S14 upon completion ofthe processing at S46-S54, or S26). As a result, the mobile terminal 40does not accept the input operation on the touch panel 46 over a periodin which the image displayed on the display region 44 a of the LCD 44 ofthe mobile terminal 40 does not match the image displayed on the displayregion 20 a of the LCD 20 of the MFP 10, reliably preventing occurrenceof a situation in which the MFP 10 erroneously recognizes an instructionthat differs from an instruction intended by the user.

The supplemental information contains the information about the presenceor absence of the next image. As a result, the mobile terminal 40 canappropriately requests the LCD data in a setting in which the LCD datais transmitted from the MFP 10 to the mobile terminal 40 on conditionthat the mobile terminal 40 requests the LCD data.

The MFP 10 transmits the LCD data to the mobile terminal 40 on conditionthat the mobile terminal 40 has requested transmission of the LCD data.This configuration allows the MFP 10 to transmit the LCD data to themobile terminal 40 at the timing when the mobile terminal 40 requiresthe LCD data. Even in a case where there are a plurality of mobileterminals 40 each configured to communicate with the MFP 10, the MFP 10can transmit the LCD data appropriately to one of the mobile terminals40 which requires the LCD data.

On condition that a screen transition is caused in the state in whichthe image data representative of the image displayed on the displayregion 20 a of the LCD 20 of the MFP 10 has not been transmitted to themobile terminal 40, the image data representative of the image beforethe screen transition is stored at S122 into the RAM 16. Accordingly,the MFP 10 can send the mobile terminal 40 the image data representativeof the image displayed before the screen transition.

The mobile terminal 40 requests transmission of the LCD data at S50 inFIG. 4 when the leaving operation is performed on the touch panel 46 ofthe mobile terminal 40 and transmitted to the MFP 10 as the type of theinput event. Here, the leaving operation is an operation for confirmingthe choice of the location on the image, and in response to the leavingoperation, the image displayed on the display region 20 a of the LCD 20of the MFP 10 is switched in many cases. Thus, the mobile terminal 40can request the LCD data at the appropriate timing.

The mobile terminal 40 is in the standby state for the predeterminedlength of time at S48 in FIG. 4 before requesting transmission of theLCD data in response to the leaving operation performed on the touchpanel 46 of the mobile terminal 40. This configuration can avoid asituation in which before the switch of the image is completed on theMFP 10, the LCD data is transmitted to the mobile terminal 40 inresponse to the request for transmission of the LCD data.

The coordinate information (i.e., (x, y) in FIG. 7A and (m, n) in FIG.7B) transmitted from the mobile terminal 40 to the MFP 10 containsstandard-size positional information for identifying the location of theinstructed point in a case where the image is of a standard size,regardless of whether the size of the image displayed on the displayregion 44 a of the LCD 44 is increased or reduced. Accordingly, the MFP10 can recognize which processing is instructed by the input operationperformed on the touch panel 46 of the mobile terminal 40, regardless ofwhether the size of the image displayed on the display region 44 a ofthe LCD 44 is increased or reduced.

The coordinate information to be transmitted from the mobile terminal 40to the MFP 10 is combined with the standard-size positional information.This configuration can simplify the coordinate information to betransmitted from the mobile terminal 40 to the MFP 10.

In the case where there are a plurality of instructed points on thedisplay region 44 a of the LCD 44 of the mobile terminal 40, the mobileterminal 40 does not transmit the type of the input event and thecoordinate information to the MFP 10, that is, in the case where thepositive decision is made at S62 in FIG. 6. This configuration can avoida situation in which unnecessary information is transmitted to the MFP10.

In a case where the press event is detected and thereafter another pressevent occurs again before the leaving event corresponding to thedetected press event, the mobile terminal 40 transfers the press cancelevent to the MFP 10 at S74 in FIG. 6. This configuration can avoid asituation in which the MFP 10 reflects erroneous information about theinput event in the case where the input operation performed on the touchpanel 46 of the mobile terminal 40 is for instructing increase orreduction in the size of the image.

The mobile terminal 40 at S66 and S68 in FIG. 6 sets not to send the MFP10 the leaving event caused after the transfer of the press cancel eventat S74 in FIG. 6, preventing unnecessary input event information to betransmitted to the MFP 10.

In the case of reception of the press cancel event, the MFP 10 executesthe canceling processing at S166 in FIG. 9. Accordingly, even in a casewhere the type of the input event transmitted from the MFP 10 isidentified later as an inappropriate one, the MFP 10 can deal with thisproblem.

In the case where the press cancel event is received, and the drag eventoccurs after the press event to be canceled, the instructed point ismoved at S164 in FIG. 9 back to the location at which the press event iscaused. As a result, the object moved in response to the drag can bereturned to the original location on the LCD 20 of the MFP 10.Accordingly, at the time when the press canceling processing (S166) isexecuted, the object can be displayed at the location at which theobject is displayed before the press event.

<Second Embodiment>

There will be next explained a second embodiment with reference todrawings, focusing mainly on differences between the first embodimentand the second embodiment.

In the above-described first embodiment, in the case where thenotification indicating that there is the next image is received fromthe MFP 10 when the mobile terminal 40 requests the MFP 10 to transmitthe LCD data, even if the mobile terminal 40 does not detect an approachor touch of the input object on the touch panel 46, the mobile terminal40 requests the next image and acquires the LCD data from the MFP 10 anddisplays an image on the display region 44 a of the LCD 44 based on theacquired LCD data. In this second embodiment, in contrast, in the casewhere the notification indicating that there is the next image isreceived, the mobile terminal 40 asks for permission to display the nextimage in advance of display of the next image.

FIG. 10 illustrates a procedure of an event reaction processing in thissecond embodiment. The CPU 52 executes this processing by executing theoperation support program stored in the flash memory 54 of the mobileterminal 40. It is noted that the same numerals as used in the flowchart in FIG. 8 are respectively used for the same processings as in theflow chart in FIG. 10.

When the CPU 52 at S90 determines that there is a next image, the CPU 52at S170 controls the LCD 44 to display an image inquiring of the user asto whether the next image is to be requested or not. The CPU 52 at S172determines whether the user as a result of the inquiry has permitted therequest of the next image or not. The CPU 52 is in the standby stateuntil the user permits the request of the next image (S172: No). Whenthe user permits the request of the next image (S172: Yes), this flowgoes to S82 for the LCD data acquiring processing.

The present embodiment described above provides the following effects inaddition to the effects obtained in the first embodiment.

In the case where the mobile terminal 40 acquiring the LCD data hasreceived the information indicating that there is the next image, as thesupplemental information, the mobile terminal 40 inquires of the user asto whether re-request of the LCD data is permitted or not, and when there-request is permitted, the mobile terminal 40 executes the LCD dataacquiring processing again (S172: Yes). This configuration allows themobile terminal 40 to execute the LCD data acquiring processingaccording to the intention of the user.

<Third Embodiment>

There will be next explained a third embodiment with reference todrawings, focusing mainly on differences between the first embodimentand the third embodiment.

In the above-described first embodiment, in view of the animationdisplay caused when the image displayed on the LCD 20 of the MFP 10 isswitched, the wait processing at S48 in FIG. 4 is executed in responseto the leaving event on the mobile terminal 40. In this thirdembodiment, in contrast, when the mobile terminal 40 requests the MFP 10to transmit the LCD data, and the MFP 10 executes the animation display,the MFP 10 does not transfer the LCD data to the mobile terminal 40until the animation display is finished.

FIG. 11 illustrates a portion of a procedure of processings to beexecuted by the MFP 10 in this third embodiment, specifically, FIG. 11illustrates changes from the processings illustrated in FIG. 9. Theseprocessings are executed by the CPU 12 of the MFP 10.

When there is no LCD data stored in the RAM 16 (S104: No), the CPU 12 atS173 determines whether the animation display is being performed or notfor switching the image displayed on the display region 20 a of the LCD20. When the animation display is not being performed (S173: No), thisflow goes to S106. When the animation display is being performed (S173:Yes), the CPU 12 at S174 sends the mobile terminal 40 an animationnotification indicating that the animation display is being performed.The CPU 12 at S176 waits for the completion of the animation display.When the animation display is completed (S176: Yes), the CPU 12 at S178notifies the mobile terminal 40 of the information indicating thecompletion of the animation, and this flow goes to S106.

FIG. 12 illustrates a procedure of processings to be executed by themobile terminal 40. The CPU 52 executes these processings by executing aremote control program stored in the flash memory 54. It is noted thatthe same numerals as used in the flow chart in FIG. 4 are respectivelyused for the same processings as in the flow chart in FIG. 12.

In this third embodiment, upon the completion of the processing at S12,the CPU 52 at S180 determines whether the CPU 52 is receiving theanimation notification from the MFP 10 or not. This processing is on theprecondition of the processings at S174 and S178 in FIG. 11. When theanimation notification is not being received (S180: No), this flow goesto S14. When the animation notification is being received (S180: Yes),the CPU 52 at S182 prohibits the acceptance of the input operation onthe touch panel 46 (i.e., the acceptance of the event) and displaysinformation about the prohibition on the display region 44 a of the LCD44, and this flow goes to S184. It is noted that the CPU 52 at S182 maybe configured to accept the input operation and not to execute aprocessing corresponding to the accepted input operation. The CPU 52 atS184 waits until the CPU 52 receives an animation completionnotification from the MFP 10 (S184: No), and when the animationcompletion notification is received (S184: Yes), this flow goes to S10.

In the present embodiment, the flow goes to S180 also in the case wherethe negative decision is made at S18, S30, or 40, or the processing iscompleted at S26, S36, or S52.

The present embodiment described above provides the following effects inaddition to the effects obtained in the first embodiment.

When the LCD data is requested, the MFP 10 does not transmit the LCDdata until the animation display is finished (at S174 in FIG. 11). Withthis configuration, a duration of the state in which the image displayedon the display region 20 a of the LCD 20 of the MFP 10 does not coincidewith the image displayed on the display region 44 a of the LCD 44 of themobile terminal 40 can be shortened as possible.

Upon receiving the animation notification indicating that the animationdisplay is being performed, from the MFP 10, the mobile terminal 40 doesnot accept the input operation on the touch panel 46. That is, themobile terminal 40 does not accept the input operation on the touchpanel 46 in the state in which the image displayed on the display region20 a of the LCD 20 of the MFP 10 does not coincide with the imagedisplayed on the display region 44 a of the LCD 44 of the mobileterminal 40. This configuration can avoid a situation in which aninstruction not intended by the user is output to the MFP 10.

<Fourth Embodiment>

There will be next explained a fourth embodiment with reference todrawings, focusing mainly on differences between the first embodimentand the fourth embodiment.

In the above-described first embodiment, even when the animation displayis performed on the display region 20 a of the LCD 20 of the MFP 10 inresponse to the switch of the image, the animation display is notperformed on the display region 44 a of the LCD 44 of the mobileterminal 40. In this fourth embodiment, in contrast, the animationdisplay is performed on the display region 44 a of the LCD 44 of themobile terminal 40.

FIG. 13 illustrates a procedure of processings to be executed by the MFP10 in the present embodiment. These processings are executed by the CPU12 of the MFP 10. It is noted that the same numerals as used in the flowchart in FIG. 9 are respectively used for the same processings as in theflow chart in FIG. 13.

As illustrated in FIG. 13, in the present embodiment, when there is noLCD data stored in the RAM 16 (S104: No), the CPU 12 at S173 determineswhether the animation display is being performed or not for switchingthe image displayed on the display region 20 a of the LCD 20. When theanimation display is not being performed (S173: No), this flow goes toS106. When the animation display is being performed (S173: Yes), the CPU12 at S190 sends the mobile terminal 40 the animation notificationindicating that the animation display is being performed.

FIG. 14 illustrates a procedure of processings to be executed by themobile terminal 40. The CPU 52 executes these processings by executingthe remote control program stored in the flash memory 54. It is notedthat the same numerals as used in the flow chart in FIG. 4 arerespectively used for the same processings as in the flow chart in FIG.14.

In the present embodiment, upon completion of the processing at S52, theCPU 52 at S200 determines whether the animation notification has beenreceived or not. This processing is on the precondition of theprocessing at S190 in FIG. 13. When the animation notification is notreceived (S200: No), this flow goes to S54.

When the animation notification is received (S200: Yes), this flowreturns to S50 to restart the LCD data acquiring processing.

The present embodiment described above provides the following effects inaddition to the effects obtained in the first embodiment.

When the animation display is performed on the display region 20 a ofthe LCD 20 of the MFP 10, the mobile terminal 40 requests the LCD dataagain. Accordingly, the animation display can be performed also on thedisplay region 44 a of the LCD 44 of the mobile terminal 40.

<Fifth Embodiment>

There will be next explained a fifth embodiment with reference todrawings, focusing mainly on differences between the first embodimentand the fifth embodiment.

In the present embodiment, the MFP 10 displays a progress bar indicatinga degree of progress of the processing, as the animation displayperformed on the display region 20 a of the LCD 20 of the MFP 10. Eachof FIGS. 15A and 15B illustrates one example of a display in which aprogress bar 70 indicating the degree of progress of the printprocessing is displayed on the display region 20 a of the LCD 20 of theMFP 10. The progress bar 70 displays the degree of progress of the printprocessing by changing the display according to the degree of progressof the print processing. The print processing has progressed in FIG. 15Bthan in FIG. 15A. In order to change or update the display of theprogress bar 70 also on the display region 44 a of the LCD 44 of themobile terminal 40 according to the degree of progress of the printprocessing, this system may be configured such that informationindicating that the progress bar 70 is displayed is incorporated intothe supplemental information, and the mobile terminal 40 periodicallyrequests transmission of the LCD data from the MFP 10 after theprocessing at S54 in FIG. 4.

Displayed on the display region 20 a of the LCD 20 of the MFP 10 is acancel button 72 for canceling the print processing before the printprocessing is completed. The LCD data acquiring processing causes thecancel button 72 to be displayed also on the display region 44 a of theLCD 44 of the mobile terminal 40. Pressing and leaving the cancel button72 displayed on the touch panel 46 of the mobile terminal 40 cancels theprint processing in the MFP 10. With this configuration, the user caninstruct the MFP 10 from the mobile terminal 40 to cancel the printprocessing.

<Sixth Embodiment>

There will be next explained a sixth embodiment, focusing mainly ondifferences between the first embodiment and the sixth embodiment.

In the third embodiment, when the animation display is performed on theMFP 10, the mobile terminal 40 does not transmit the touch panel eventto the MFP 10 until the animation display is completed. In this sixthembodiment, however, the mobile terminal 40 transmits the touch panelevent only in the case of particular animation.

Specifically, when the animation notification is being received at S180in FIG. 12 in the third embodiment (S180: Yes) and when the CPU 52,based on the supplemental information acquired at S12, determines thatthe type of animation displayed on the MFP is the progress barillustrated in FIGS. 15A and 15B, the flow goes to S14, and when thetype of animation is not the progress bar, the flow goes to S182. It isnoted that the MFP 10 incorporates information for identifying the typeof animation displayed on the MFP 10, into the supplemental informationto be transmitted to the mobile terminal 40.

In view of the above, the LCD 44 is one example of a display device or aterminal-side display device. The LCD 20 is one example of anapparatus-side display device. The touch panel 46 is one example of aninput device or a terminal-side input device. The touch panel 22 is oneexample of an apparatus-side input device. The communication device 42is one example of a communication device. The CPU 52 is one example of acontroller. The MFP 10 is one example of a processing apparatus. The RAM16 is one example of a storage device. The LCD data is one example ofinstruction image data.

<Other Embodiments>

While the embodiments of the present invention have been describedabove, it is to be understood that the invention is not limited to thedetails of the illustrated embodiments, but may be embodied with variouschanges and modifications, which may occur to those skilled in the art,without departing from the spirit and scope of the invention. Eachembodiment may include at least a part of the configuration of anotherembodiment and may be at least partly replaced with the configuration ofanother embodiment. Alternative embodiments of the present inventionwill be explained below.

In the above-described embodiments, the mobile terminal 40 at S48 inFIG. 4 executes the wait processing corresponding to a length of timerequired for the animation display for switching the image displayed onthe display region 20 a of the LCD 20 of the MFP 10, but the presentinvention is not limited to this configuration. For example, when themobile terminal 40 requests the MFP 10 to transmit the LCD data, the MFP10 may transmit the LCD data after waiting for the completion of theanimation display on the MFP 10. In this configuration, the mobileterminal 40 preferably does not accept the input on the touch panel 46until the mobile terminal 40 requests the LCD data, then receives theLCD data, and finishes switching the image displayed on the displayregion 44 a of the LCD 44.

In the above-described embodiments, the mobile terminal 40 requests theMFP 10 to transmit the LCD data when the image displayed on the displayregion 20 a of the LCD 20 of the MFP 10 does not match the imagedisplayed on the display region 44 a of the LCD 44 of the mobileterminal 40, but the present invention is not limited to thisconfiguration. For example, when the MFP 10 determines that the imagesdo not coincide with each other, the MFP 10 may voluntarily transmit theLCD data to the mobile terminal 40. When the MFP 10 further transmitsthe screen ID to the mobile terminal 40 in this case, the mobileterminal 40 can recognize that the image data transmitted from the MFP10 differs from the image being displayed on the display region 44 a ofthe LCD 44.

The timing of transmission of the screen ID from the mobile terminal 40to the MFP 10 is not limited to the timing of transmission of thecoordinate information and the like (at S24) in response to the pressevent. For example, the mobile terminal 40 may periodically transmit thescreen ID to the MFP 10 in a predetermined cycle.

In FIG. 2, considering the displacement of the object indicating thelocation at which the instructed point should be located for instructingthe processing to be executed by the MFP 10 when the screen itself isswitched, the mobile terminal 40 cannot accept the input operationperformed on the touch panel 46 when the screen itself is switched.However, the present invention is not limited to this configuration. Forexample, the mobile terminal 40 may be configured not to switch theentire image displayed on the display region 20 a of the LCD 20 of theMFP 10 but not to accept the input operation performed on the touchpanel 46 of the mobile terminal 40 when only the object indicating thelocation at which the instructed point should be located for instructingthe processing to be executed by the MFP 10 is displaced.

The request of the LCD data and the request of the supplementalinformation are separately output from the mobile terminal 40 to the MFP10 in the above-described embodiments, but the present invention is notlimited to this configuration. For example, the request of the LCD datamay cause the MFP 10 to transmit the LCD data and the supplementalinformation corresponding thereto.

The input operation performed on the touch panel 46 of the mobileterminal 40 is not accepted at S46 and S54 in FIG. 4 within the periodof the animation display on the MFP 10 in the above-describedembodiments, but the present invention is not limited to thisconfiguration. For example, the mobile terminal 40 may be configured tochange the display of the display region 44 a of the LCD 44 in responseto the input operation and to display information indicating that theinput operation is disabled, on the display region 44 a of the LCD 44,and not to send the MFP 10 the coordinate information and the inputevent relating to the input operation.

The input operation on the touch panel 46 is not accepted within theperiod of, e.g., the wait processing at S48 in the processing FIG. 4 inthe above-described embodiments. Even within this period, however, thetouch panel 46 may accept an input operation not relating to theinstruction provided for the MFP 10, and one example of such an inputoperation is the pinch-out operation for instructing enlargement of thescreen.

In the method in FIG. 7B, the reference point and the predeterminedpoint are respectively set at the drawing start coordinates (X, Y) andthe drawing end coordinates (Xe, Ye), and the legnth of each of thecoordinate components from the reference point to the predeterminedpoint is used as the standard-size positional information foridentifying the location of the instructed point in the case where theimage is of a standard size, but the present invention is not limited tothis configuration. For example, the predetermined point is set by thecoordinate component of the midpoint of the image in the X-axisdirection and the coordinate component of the midpoint of the image inthe Y-axis direction. Also in this configuration, the length of each ofthe coordinate components from the reference point to the predeterminedpoint may be used as the standard-size positional information foridentifying the location of the instructed point in the case where theimage is of a standard size.

Regardless of whether the size of the image is increased or reduced, thecoordinate information to be transmitted from the mobile terminal 40 tothe MFP 10 is not limited to coordinate information integrated with thestandard-size positional information for identifying the location of theinstructed point in the case where the image is of a standard size (theenlargement ratio k in FIG. 7A and (Xe−Y, Ye−Y) in FIG. 7B). Forexample, instead of the method illustrated in FIG. 7A, the coordinatecomponents (A−X, B−Y) provided with the drawing start coordinates as theorigin point and the enlargement ratio k as the standard-size positionalinformation may be separately transmitted to the MFP 10. Also in thisconfiguration, the MFP 10 can identify the location of the instructedpoint in the case where the image is of a standard size. Also, theinformation transmitted from the mobile terminal 40 to the MFP 10 is notlimited to information containing the standard-size positionalinformation. That is, in a case where the processing for increasing orreducing the size of the image is not executed on the mobile terminal40, even when the information transmitted from the mobile terminal 40 tothe MFP 10 does not contain the standard-size positional information,the MFP 10 can identify the location of the instructed point in the casewhere the image is of a standard size.

The method of requesting the MFP 10 to transmit the LCD data is notlimited to a method using the leaving operation on the touch panel 46 asa trigger. For example, the method may use the press operation on thetouch panel 46 as a trigger.

The method of requesting the MFP 10 to transmit the LCD data is notlimited to a method using the input event on the touch panel 46 as atrigger. For example, the method of requesting the MFP 10 to transmitthe LCD data may be a method of requesting the LCD data in a cycle of apredetermined length of time. In this configuration, this system may beconfigured such that the mobile terminal 40 further transmits the screenID to the MFP 10, and the MFP 10 transfers the LCD data to the mobileterminal 40 on condition that the screen ID of the image displayed onthe display region 44 a of the LCD 44 of the mobile terminal 40 does notcoincide with the screen ID of the image displayed on the display region20 a of the LCD 20 of the MFP 10.

The MFP 10 may not include the LCD 20 or the touch panel 22. In a casewhere the MFP 10 not including the LCD 20 or the touch panel 22 isconfigured to send the mobile terminal 40 image data representative ofthe image for guiding the input operation on the touch panel 46, theuser can operate the MFP 10 by performing the input operation on thetouch panel 46 of the mobile terminal 40. In this configuration, the MFP10 cannot set the image to be displayed on the display region 20 a ofthe LCD 20, but in a case where the MFP 10 sets images for guidingoperations acceptable on the MFP 10, the MFP 10 can execute, e.g., theprocessings at S128-S140 in FIG. 9 for comparing the screen IDs witheach other. In this configuration, examples of the situation in whichthe screen IDs do not match each other as a result of the comparisoninclude: a case where a noise occurs in the mobile terminal 40, the MFP10, or communication between the mobile terminal 40 and the MFP 10; anda case where a malfunction occurs in the mobile terminal 40 due to atemporal power shortage of the mobile terminal 40.

The input devices (i.e., the terminal-side input device and theapparatus-side input device) are not limited to the touch panel 22, 46.For example, the input devices may be devices configured to use imagerecognition to detect the input operations on the LCDs 20, 44 such aspressing, leaving, and dragging.

The communication between the mobile terminal 40 and the MFP 10 when themobile terminal 40 controls the MFP 10 is not limited to the Wi-Ficommunication. For example, the communication may be communication usingthe telephone network communication device 50.

The device or apparatus (i.e., the processing apparatus) controllable bythe mobile terminal 40 is not limited to the MFP 10. For example, themobile terminal 40 may control a home energy management system or deviceequipped in a home. In this configuration, the use of the telephonenetwork communication device 50 allows a user to activate devices suchas an air-conditioning equipment before getting home from a place farfrom his or her home and allows a user to instruct the system to useenergy generated by solar panels for boiling water with a higherpriority, for example. In a case where the energy management device isequipped with a touch panel, the user can provide instructions from theoutside of his or her home with operations corresponding to operationson the touch panel of the energy management device in his or her home.

The communication terminal is not limited to a device having functionsother than the functions relating to the operations for the processingapparatus, e.g., the MFP 10. That is, the communication terminal is notlimited to a device like a smartphone having various functions includingthe function of conversation over the telephone network. However, thecommunication terminal may be a device specialized for the operationsfor the processing apparatus.

The processings, i.e., software processings executed by the CPU 52executing the operation support program in the above-describedembodiments may be partly or completely replaced with hardwareprocessings executed by, e.g., an ASIC.

The processings to be executed by the processing apparatus include theprocessing for switching the screen in addition to the processings forexecuting the various functions such as the processing for executing thecopying function, the processing for executing the facsimile function,and the processing for executing the scanning function. For example, theprocessings to be executed by the processing apparatus include aprocessing in which when the Fax icon is chosen on the screenillustrated in FIG. 2A, the screen or image is switched to the screen orimage illustrated in FIG. 2C.

The screen ID for the image data representative of the image beingdisplayed on the MFP 10 is one example of identification informationabout the instruction image data representative of an instruction imagecurrently displayed. As illustrated in FIG. 3, the screen IDsrespectively correspond to the image data. Accordingly, the screen IDsare assigned also to image data representative of images not displayedon the MFP 10.

What is claimed is:
 1. A non-transitory storage medium storing aplurality of instructions executable by a computer of a communicationterminal, the communication terminal comprising: a display; an inputdevice configured to detect an instructed point which is a point on thedisplay and at which an instruction is provided with an input object,the input device being configured to accept a plurality of types ofinput gestures as a type of an input event; and a communication deviceconfigured to communicate with a processing apparatus, when executed bythe computer, the plurality of instructions causing the communicationterminal to perform: receiving instruction image data from theprocessing apparatus, the instruction image data being for displaying,on the display, a plurality of instruction images, each image forinstructing a processing to be executed by the processing apparatus;receiving a plurality of image IDs from the processing apparatus, eachof the plurality of image IDs being for identifying a corresponding oneof the plurality of instruction images, displaying a first instructionimage on the display based on the received instruction image data, thefirst instruction image being one of the plurality of instructionimages; identifying a type of input gesture performed on the inputdevice from among the plurality of types of input gestures, in responseto an instruction provided on the input device with the input object ina state in which the first instruction image is displayed; andtransmitting, to the processing apparatus, (i) the identified type ofinput gesture from among the plurality of types of input gestures, (ii)coordinate information about the instructed point on the display whichis detected by the input device in a state in which the firstinstruction image is displayed, and (iii) a first image ID identifyingthe first instruction image among the plurality of instruction images,the first image ID being one of the plurality of image IDs.
 2. Thenon-transitory storage medium according to claim 1, wherein whenexecuted by the computer, the plurality of instructions cause thecommunication terminal to perform: making a request of the processingapparatus for transmission of the instruction image data; and receivingthe instruction image data transmitted from the processing apparatus inresponse to the request of the processing apparatus for transmission ofthe instruction image data.
 3. The non-transitory storage mediumaccording to claim 1, wherein when executed by the computer, theplurality of instructions cause the communication terminal to perform:receiving, from the processing apparatus, a difference notificationindicating that the first image ID about the instruction image datarepresentative of the first instruction image currently displayed on thedisplay of the communication terminal differs from a second image IDabout image data representative of an image currently displayed on theprocessing apparatus; and displaying, on the display, an imageindicating that the first image ID about the instruction image datarepresentative of the first instruction image currently displayed on thedisplay of the communication terminal differs from the second image IDabout the image data representative of the image currently displayed onthe processing apparatus, in response to the reception of the differencenotification.
 4. The non-transitory storage medium according to claim 1,wherein when executed by the computer, the plurality of instructionscause the communication terminal to perform: receiving, from theprocessing apparatus, a difference notification indicating that thefirst image ID about the instruction image data representative of thefirst instruction image currently displayed on the display of thecommunication terminal differs from a second image ID about image datarepresentative of an image currently displayed on the processingapparatus; and requesting the processing apparatus to transmit the imagedata representative of the image currently displayed on the processingapparatus, in response to the reception of the difference notification.5. The non-transitory storage medium according to claim 1, wherein whenexecuted by the computer, the plurality of instructions cause thecommunication terminal not to transmit, to the processing apparatus, atype of input gesture on the input device which is performed within atleast a period extending from a request of the processing apparatus fortransmission of the instruction image data, to the reception of theinstruction image data transmitted in response to request.
 6. Thenon-transitory storage medium according to claim 2, wherein whenexecuted by the computer, the plurality of instructions cause thecommunication terminal to perform: receiving a request instruction whichis an instruction transmitted from the processing apparatus to thecommunication terminal and which instructs the communication terminal torequest the instruction image data from the processing apparatus; andmaking the request of the processing apparatus for transmission of theinstruction image data, in response to the reception of the requestinstruction.
 7. The non-transitory storage medium according to claim 1,wherein the processing apparatus comprises: an apparatus-side display;and an apparatus-side input device configured to detect an instructedpoint which is a point on the apparatus-side display and at which aninstruction is provided with an input object, and wherein the firstinstruction image to be displayed based on the received instructionimage data is identical to an image to be displayed on theapparatus-side display.
 8. The non-transitory storage medium accordingto claim 2, wherein the request of the processing apparatus fortransmission of the instruction image data is made based on transmissionof the identified type of input gesture to the processing apparatus. 9.The non-transitory storage medium according to claim 8, wherein theinput event comprises a choice of a point on the display and aconfirmation of the choice of the point, and wherein the request of theprocessing apparatus for transmission of the instruction image data ismade based on transmission of the confirmation of the choice of thepoint to the processing apparatus.
 10. A non-transitory storage mediumstoring a plurality of instructions executable by a computer of acommunication terminal, the communication terminal comprising: adisplay; an input device configured to detect an instructed point whichis a point on the display and at which an instruction is provided withan input object; and a communication device configured to communicatewith a processing apparatus, when executed by the computer, theplurality of instructions causing the communication terminal to perform:receiving instruction image data from the processing apparatus, theinstruction image data being for displaying, on the display, aninstruction image for instructing a processing to be executed by theprocessing apparatus; displaying the instruction image on the displaybased on the received instruction image data; identifying an input eventon the input device in response to an instruction provided on the inputdevice with the input object in a state in which the instruction imageis displayed; and transmitting, to the processing apparatus, theidentified input event and coordinate information about the instructedpoint on the display which is detected by the input device in a state inwhich the instruction image is displayed, wherein when executed by thecomputer, the plurality of instructions cause the communication terminalto perform: making a request of the processing apparatus fortransmission of the instruction image data; and receiving theinstruction image data transmitted from the processing apparatus inresponse to the request of the processing apparatus for transmission ofthe instruction image data, wherein the request of the processingapparatus for transmission of the instruction image data is made basedon transmission of the input event to the processing apparatus, whereinthe input event comprises a choice of a point on the display and aconfirmation of the choice of the point, wherein the request of theprocessing apparatus for transmission of the instruction image data ismade based on transmission of the confirmation of the choice of thepoint to the processing apparatus, wherein the processing apparatuscomprises: an apparatus-side display; and an apparatus-side input deviceconfigured to detect an instructed point which is a point on theapparatus-side display and at which an instruction is provided with aninput object, wherein the instruction image to be displayed based on thereceived instruction image data is identical to an image to be displayedon the apparatus-side display, and wherein when executed by thecomputer, the plurality of instructions cause the communication terminalto wait for making the request of the processing apparatus fortransmission of the instruction image data, for a predetermined periodafter the transmission of the confirmation of the choice of the point tothe processing apparatus.
 11. The non-transitory storage mediumaccording to claim 10, wherein when executed by the computer, theplurality of instructions cause the communication terminal not totransmit the input event having occurred within the predeterminedperiod, to the processing apparatus.
 12. A communication terminal,comprising: a display; an input device configured to detect aninstructed point which is a point on the display and at which aninstruction is provided with an input object, the input device beingconfigured to accept a plurality of types of input gestures as a type ofan input event; a communication device configured to communicate with aprocessing apparatus; and a controller configured to: receiveinstruction image data from the processing apparatus, the instructionimage data being for displaying, on the display, a plurality ofinstruction images, each image for instructing a processing to beexecuted by the processing apparatus; receive a plurality of IDs fromthe processing apparatus, each of the plurality of IDs being foridentifying a corresponding one of the plurality of instruction images;display a first instruction image on the display based on the receivedinstruction image data, the first instruction image being one of theplurality of instruction images; identify a type of input gestureperformed on the input device from among the plurality of types of inputgestures, in response to an instruction provided on the input devicewith the input object in a state in which the first instruction image isdisplayed; and transmit, to the processing apparatus, (i) the identifiedtype of input gesture from among the plurality of types of inputgestures, (ii) coordinate information about the instructed point on thedisplay which is detected by the input device in a state in which thefirst instruction image is displayed, and (iii) a first image IDidentifying the first instruction image among the plurality ofinstruction images, the first image ID being one of the plurality ofimage IDs, the controller being configured to execute a control using atleast one of a software processing and a hardware processing.
 13. Thecommunication terminal according to claim 12, wherein the controller isconfigured to: make a request of the processing apparatus fortransmission of the instruction image data; and receive the instructionimage data transmitted from the processing apparatus in response to therequest of the processing apparatus for transmission of the instructionimage data.
 14. A processing apparatus controllable by a communicationterminal, the communication terminal comprising: a display; an inputdevice configured to detect an instructed point which is a point on thedisplay and at which an instruction is provided with an input object,the input device being configured to accept a plurality of types ofinput gestures as a type of an input event; and a communication deviceconfigured to communicate with the processing apparatus, the processingapparatus comprising a controller configured to: transmit instructionimage data to the communication terminal, the instruction image databeing for displaying, on the display, a plurality of instruction images,each image for instructing a processing to be executed by the processingapparatus; transmit a plurality of image IDs to the communicationterminal, each of the plurality of image IDs being for identifying acorresponding one of the plurality of instruction images; receive, fromthe communication terminal, (i) a type of input gesture performed on theinput device among the plurality of types of input gestures, (ii)coordinate information about the instructed point on the display whichis detected by the input device, and (iii) a first image ID identifyingthe first instruction image among the plurality of instruction images,the first image ID being one of the plurality of image IDs; and executea processing based on the received type of input gesture, coordinateinformation and the first image ID.
 15. The processing apparatusaccording to claim 14, wherein the controller is configured to: acquire,from the communication terminal, the first image ID about instructionimage data representative of the first instruction image currentlydisplayed on the display of the communication terminal; determinewhether the acquired the first image ID coincides with a second image IDabout image data representative of an image currently displayed on theprocessing apparatus; and execute a processing based on the receivedtype of input gesture when the acquired the first image ID coincideswith the second image ID about the image data representative of theimage currently displayed on the processing apparatus.
 16. Theprocessing apparatus according to claim 15, wherein the controller isconfigured to transmit, to the communication terminal, informationindicating that the acquired first image ID does not coincide with thesecond image ID about the image data representative of the imagecurrently displayed on the processing apparatus, when the acquired firstimage ID does not coincide with the second image ID about the image datarepresentative of the image currently displayed on the processingapparatus.
 17. The processing apparatus according to claim 14, furthercomprising an apparatus-side display; and an apparatus-side input deviceconfigured to detect an instructed point which is a point on theapparatus-side display and at which an instruction is provided with aninput object, wherein the controller is configured to: receive, from thecommunication terminal, a request of the processing apparatus fortransmission of the instruction image data; and transmit image datarepresentative of an image currently displayed on the apparatus-sidedisplay, to the communication terminal, based on the reception of therequest of the processing apparatus for transmission of the instructionimage data.
 18. The processing apparatus according to claim 14, furthercomprising an apparatus-side display; and an apparatus-side input deviceconfigured to detect an instructed point which is a point on theapparatus-side display and at which an instruction is provided with aninput object, wherein the controller is configured to transmit, to thecommunication terminal, image data representative of an image currentlydisplayed on the apparatus-side display.
 19. The processing apparatusaccording to claim 14, wherein the controller is configured to: receivea request of the processing apparatus for transmission of theinstruction image data; and transmit the instruction image data to thecommunication terminal in response to the received request of theprocessing apparatus for transmission of the instruction image data. 20.A processing apparatus controllable by a communication terminal, thecommunication terminal comprising: a display; an input device configuredto detect an instructed point which is a point on the display and atwhich an instruction is provided with an input object; and acommunication device configured to communicate with the processingapparatus, the processing apparatus comprising a controller configuredto: transmit instruction image data to the communication terminal, theinstruction image data being for displaying, on the display, aninstruction image for instructing a processing to be executed by theprocessing apparatus; receive, from the communication terminal, an inputevent on the input device and coordinate information about theinstructed point on the display which is detected by the input device;and execute a processing based on the received input event andcoordinate information, wherein the processing apparatus furthercomprises an apparatus-side display; and an apparatus-side input deviceconfigured to detect an instructed point which is a point on theapparatus-side display and at which an instruction is provided with aninput object, wherein the controller is configured to: receive, from thecommunication terminal, a request of the processing apparatus fortransmission of the instruction image data; and transmit image datarepresentative of an image currently displayed on the apparatus-sidedisplay, to the communication terminal, based on the reception of therequest of the processing apparatus for transmission of the instructionimage data, wherein the processing apparatus further comprises a storagedevice, wherein the controller is configured to, when an image displayedon the apparatus-side display is switched in a state in which an imagebased on image data having not been transmitted to the communicationterminal is displayed on the apparatus-side display, store, into thestorage device, the image data representative of the image displayedbefore the image is switched.
 21. The processing apparatus according toclaim 20, wherein the controller is configured to, based on a receptionof the request of the transmission of the instruction image data in astate in which the image data is stored in the storage device before theswitch, transmit the image data stored before the switch, to thecommunication terminal.